<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<!-- saved from url=(0014)about:internet -->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

<title>Chapter 6: Exercise 4</title>

<base target="_blank"/>

<style type="text/css">
body, td {
   font-family: sans-serif;
   background-color: white;
   font-size: 12px;
   margin: 8px;
}

tt, code, pre {
   font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace;
}

h1 { 
   font-size:2.2em; 
}

h2 { 
   font-size:1.8em; 
}

h3 { 
   font-size:1.4em; 
}

h4 { 
   font-size:1.0em; 
}

h5 { 
   font-size:0.9em; 
}

h6 { 
   font-size:0.8em; 
}

a:visited {
   color: rgb(50%, 0%, 50%);
}

pre {	
   margin-top: 0;
   max-width: 95%;
   border: 1px solid #ccc;
}

pre code {
   display: block; padding: 0.5em;
}

code.r {
   background-color: #F8F8F8;
}

table, td, th {
  border: none;
}

blockquote {
   color:#666666;
   margin:0;
   padding-left: 1em;
   border-left: 0.5em #EEE solid;
}

hr {
   height: 0px;
   border-bottom: none;
   border-top-width: thin;
   border-top-style: dotted;
   border-top-color: #999999;
}

@media print {
   * { 
      background: transparent !important; 
      color: black !important; 
      filter:none !important; 
      -ms-filter: none !important; 
   }

   body { 
      font-size:12pt; 
      max-width:100%; 
   }
       
   a, a:visited { 
      text-decoration: underline; 
   }

   hr { 
      visibility: hidden;
      page-break-before: always;
   }

   pre, blockquote { 
      padding-right: 1em; 
      page-break-inside: avoid; 
   }

   tr, img { 
      page-break-inside: avoid; 
   }

   img { 
      max-width: 100% !important; 
   }

   @page :left { 
      margin: 15mm 20mm 15mm 10mm; 
   }
     
   @page :right { 
      margin: 15mm 10mm 15mm 20mm; 
   }

   p, h2, h3 { 
      orphans: 3; widows: 3; 
   }

   h2, h3 { 
      page-break-after: avoid; 
   }
}

</style>



<!-- MathJax scripts -->
<script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>



</head>

<body>
<h1>Chapter 6: Exercise 4</h1>

<h3>a</h3>

<p>(iii) Steadily increase: As we increase \( \lambda \) from \( 0 \), all \( \beta \) &#39;s decrease from their least square estimate values to \( 0 \). Training error for full-blown-OLS \( \beta \) s is the minimum and it steadily increases as \( \beta \) s are reduced to \( 0 \).</p>

<h3>b</h3>

<p>(ii) Decrease initially, and then eventually start increasing in a U shape: When \( \lambda = 0 \), all  \( \beta \) s have their least square estimate values. In this case, the model tries to fit hard to training data and hence test RSS is high. As we increase \( \lambda \), \( beta \) s start reducing to zero and some of the overfitting is reduced. Thus, test RSS initially decreases. Eventually, as \( beta \) s approach \( 0 \), the model becomes too simple and test RSS increases.</p>

<h3>c</h3>

<p>(iv) Steadily decreases: When \( \lambda = 0 \), the \( \beta \) s have their least square estimate values. The actual estimates heavily depend on the training data and hence variance is high. As we increase \( \lambda \), \( \beta \) s start decreasing and model becomes simpler. In the limiting case of \( \lambda \) approaching infinity, all \( beta \) s reduce to zero and model predicts a constant and has no variance. </p>

<h3>d</h3>

<p>(iii) Steadily increases: When \( \lambda = 0 \), \( \beta \) s have their least-square estimate values and hence have the least bias. As \( \lambda \) increases, \( \beta \) s start reducing towards zero, the model fits less accurately to training data and hence bias increases. In the limiting case of \( \lambda \) approaching infinity, the model predicts a constant and hence bias is maximum.</p>

<h4>e</h4>

<p>(v) Remains constant: By definition, irreducible error is model independent and hence irrespective of the choice of \( \lambda \), remains constant.</p>

</body>

</html>

